Skip to content

fix(#5436): move header config to user-edited YAML file#5442

Open
fullsend-ai-coder[bot] wants to merge 5 commits into
mainfrom
agent/5436-fix-header-passthrough
Open

fix(#5436): move header config to user-edited YAML file#5442
fullsend-ai-coder[bot] wants to merge 5 commits into
mainfrom
agent/5436-fix-header-passthrough

Conversation

@fullsend-ai-coder

@fullsend-ai-coder fullsend-ai-coder Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

Fix --header key=value flags being silently ignored by otelcol-contrib in hack/upload-traces.sh. The Collector reads headers from its YAML config, not from env vars. Instead of generating YAML dynamically from CLI flags (which introduced escaping, injection, and delimiter bugs across multiple review rounds), restore the static config template and let users edit it directly.

Related Issue

Fixes #5436

Changes

  • Remove --header flag and all header parsing/merging/escaping logic
  • Restore hack/upload-traces-otelcol-config.yaml with a documented headers: section for users to fill in
  • Add confmap escaping for endpoint and include-pattern env vars to prevent ${scheme:...} expansion
  • Add a yq-based check that prompts for confirmation when no headers are configured
  • Update usage text to point users at the config file

Testing

  • make lint passes (stage changes first, then run)
  • Tests added/updated for new or modified logic

Checklist

  • PR title follows Conventional Commits (correct type, ! for breaking changes)
  • Commits are signed off (DCO) — human and human-directed agent sessions only
  • I wrote this contribution myself and can explain all changes in it

The otelcol-contrib Collector reads headers from its YAML config
(exporters.otlphttp.headers), not from OTEL_EXPORTER_OTLP_HEADERS
which is an OpenTelemetry SDK env var. The script was setting
OTEL_EXPORTER_OTLP_HEADERS, which the Collector silently ignored,
so --header flags like x-mlflow-experiment-id had no effect.

Generate a runtime config file that injects parsed headers into
the otlphttp exporter's headers block as YAML key-value pairs.
The script still merges --header flags with any pre-existing
OTEL_EXPORTER_OTLP_HEADERS env var for backwards compatibility.
Header values are double-quoted for YAML safety. The temp config
is cleaned up on exit via trap.

Note: make lint could not run in sandbox (pre-commit failed due
to network restrictions). Manual verification of YAML output
confirmed correct structure for: no headers, single header,
multiple headers, and values containing YAML special characters.

Closes #5436
@fullsend-ai-coder
fullsend-ai-coder Bot requested a review from a team as a code owner July 22, 2026 07:23
@fullsend-ai-coder fullsend-ai-coder Bot added the ready-for-review Agent PR ready for human review label Jul 22, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Jul 22, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 7:25 AM UTC · Completed 7:38 AM UTC
Commit: 0edd109 · View workflow run →

@github-actions

github-actions Bot commented Jul 22, 2026

Copy link
Copy Markdown

Site preview

Preview: https://4bd1bd0e-site.fullsend-ai.workers.dev

Commit: b6d72c2b69d5559491dda21c12193144cc2cee4b

@fullsend-ai-review

Copy link
Copy Markdown

Review

Findings

Medium

  • [stale-reference] hack/upload-traces.sh:74 — The script still validates that CONFIG_TEMPLATE (the static YAML file upload-traces-otelcol-config.yaml) exists on disk, but the generated RUNTIME_CONFIG is now used instead. The CONFIG_TEMPLATE variable (line 5) and its validation are dead code that will produce a misleading error if the now-unused template file is later removed.
    Remediation: Remove the CONFIG_TEMPLATE variable (line 5) and its existence check (lines 74–75), or remove only the validation if the template is kept for reference.

  • [YAML-injection] hack/upload-traces.sh:127 — Header keys and values are written into the generated YAML config with insufficient sanitization. Keys have no sanitization at all; values only escape double quotes but not embedded newlines (which break out of YAML double-quoted strings) or YAML backslash escape sequences (\n, \t, etc., which are interpreted by YAML parsers). The OTEL_EXPORTER_OTLP_HEADERS env var could be set by CI automation, making this a viable injection path in shared environments.
    Remediation: Validate keys match [a-zA-Z0-9_-]+ at parse time, strip newline characters from values, and consider single-quoted YAML scalars for values (which have no escape sequences).

Low

  • [edge-case] hack/upload-traces.sh:123 — Header values containing commas are incorrectly split due to the comma-separated join/split approach (MERGED_HEADERS joined with ,, then IFS=',' read -ra). This is a pre-existing constraint inherited from the OTEL_EXPORTER_OTLP_HEADERS comma-separated format, not a regression introduced by this PR.

Labels: PR fixes a bug in a developer utility script (hack/upload-traces.sh)

@fullsend-ai-review fullsend-ai-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the review comment for full details.


Note: The following inline comments could not be posted on the diff (GitHub returned 422) and are included here instead:

  • hack/upload-traces.sh (file-level): Line 74 · [medium] stale-reference

The script still validates that CONFIG_TEMPLATE (the static YAML file upload-traces-otelcol-config.yaml) exists on disk, but the generated RUNTIME_CONFIG is now used instead. The CONFIG_TEMPLATE variable (line 5) and its validation are dead code that will produce a misleading error if the now-unused template file is later removed.

Suggested fix: Remove the CONFIG_TEMPLATE variable (line 5) and its existence check (lines 74-75), or remove only the validation if the template is kept for reference.

  • hack/upload-traces.sh:127: [medium] YAML-injection

Header keys and values are written into the generated YAML config with insufficient sanitization. Keys have no sanitization at all; values only escape double quotes but not embedded newlines (which break out of YAML double-quoted strings) or YAML backslash escape sequences. The OTEL_EXPORTER_OTLP_HEADERS env var could be set by CI automation, making this a viable injection path in shared environments.

Suggested fix: Validate keys match [a-zA-Z0-9_-]+ at parse time, strip newline characters from values, and consider single-quoted YAML scalars for values (which have no escape sequences).

  • hack/upload-traces.sh:123: [low] edge-case

Header values containing commas are incorrectly split due to the comma-separated join/split approach. This is a pre-existing constraint inherited from the OTEL_EXPORTER_OTLP_HEADERS comma-separated format, not a regression introduced by this PR.

@fullsend-ai-review fullsend-ai-review Bot added requires-manual-review Review requires human judgment bug labels Jul 22, 2026
@codecov

codecov Bot commented Jul 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Signed-off-by: Hector Martinez <hemartin@redhat.com>
@rh-hemartin

Copy link
Copy Markdown
Member

/fs-fix

@fullsend-ai-coder

fullsend-ai-coder Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Finished Fix · ✅ Success · Started 11:51 AM UTC · Completed 11:56 AM UTC
Commit: 7ada4e0 · View workflow run →

- Remove unused SCRIPT_DIR, CONFIG_TEMPLATE variable and its file
  existence check (dead code after switch to RUNTIME_CONFIG)
- Validate header keys match [a-zA-Z0-9_-]+ to prevent YAML injection
- Strip newline/carriage-return characters from header values
- Switch to single-quoted YAML scalars (no backslash escape sequences)

Addresses review feedback on #5442
@fullsend-ai-coder

Copy link
Copy Markdown
Contributor Author

🔧 Fix agent — iteration 1 (human-triggered)

Addressed 2 of 3 review findings. Removed dead CONFIG_TEMPLATE code (stale-reference) and hardened YAML config generation against injection (key validation, newline stripping, single-quoted scalars). Disagreed with 1 finding (comma-in-value edge case) as it is pre-existing and out of scope per the reviewer's own assessment.

Fixed (2):

  1. stale-reference: dead CONFIG_TEMPLATE variable and validation (hack/upload-traces.sh): Removed SCRIPT_DIR (line 4), CONFIG_TEMPLATE (line 5), and its existence check (lines 75-76). All were dead code since the script now generates and uses RUNTIME_CONFIG instead of the static template.
  2. YAML-injection: insufficient header key/value sanitization (hack/upload-traces.sh): Added key validation (must match [a-zA-Z0-9_-]+), newline/CR stripping from values, and switched from double-quoted to single-quoted YAML scalars (which have no backslash escape sequences). Single quotes in values are escaped as '' per YAML spec.

Disagreed (1):

  1. edge-case: header values containing commas split incorrectly: The reviewer explicitly acknowledged this is a pre-existing constraint inherited from the OTEL_EXPORTER_OTLP_HEADERS comma-separated format, not a regression introduced by this PR. Fixing it would require changing the header merging contract, which is out of scope.

Tests: passed

Updated by fullsend fix agent

@rh-hemartin

Copy link
Copy Markdown
Member

/fs-fix is the committed configuration file required anymore? Maybe we can remove it.

@fullsend-ai-coder

fullsend-ai-coder Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Finished Fix · ✅ Success · Started 12:00 PM UTC · Completed 12:05 PM UTC
Commit: 7ada4e0 · View workflow run →

@fullsend-ai-coder

Copy link
Copy Markdown
Contributor Author

🔧 Fix agent — iteration 2 (human-triggered)

Removed the now-unused hack/upload-traces-otelcol-config.yaml config template. The PR already removed all references to it from upload-traces.sh (CONFIG_TEMPLATE variable and file-existence check) and generates the config at runtime instead. No other files reference the template.

Fixed (1):

  1. remove unused committed configuration file (hack/upload-traces-otelcol-config.yaml): Deleted hack/upload-traces-otelcol-config.yaml. The PR switched upload-traces.sh to generate the Collector config at runtime, so the committed template is no longer referenced anywhere and is dead code.

Tests: passed

Updated by fullsend fix agent

fullsend-ai-coder Bot added a commit that referenced this pull request Jul 22, 2026
The committed upload-traces-otelcol-config.yaml is no longer referenced
after the script was updated to generate the Collector config at runtime.
Remove the dead file.

Addresses review feedback on #5442

@waynesun09 waynesun09 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated multi-model review pass (2 independent reviewers completed; a third did not finish in time). Findings below are newly surfaced against the current head commit — the previously-flagged dead CONFIG_TEMPLATE reference and the original YAML-escaping gaps (missing key validation, raw double-quote-only escaping) are already fixed as of the latest commit and are not re-flagged here. A previously-discussed header-value comma-splitting limitation (inherited from the OTEL_EXPORTER_OTLP_HEADERS env-var convention, not a regression from this PR) is likewise not re-flagged. Posting 1 HIGH and 2 MEDIUM findings.

Comment thread hack/upload-traces.sh Outdated
Comment thread hack/upload-traces.sh Outdated
Comment thread hack/upload-traces.sh Outdated
The committed upload-traces-otelcol-config.yaml is no longer referenced
after the script was updated to generate the Collector config at runtime.
Remove the dead file.

Addresses review feedback on #5442

Signed-off-by: Hector Martinez <hemartin@redhat.com>

@waynesun09 waynesun09 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recheck pass against head 2843acf6 (following up on the prior review round at an earlier commit).

Verified fixed:

  • HIGH (cleartext header/secret printing to stdout): confirmed removed. The echo "headers: $MERGED_HEADERS" / cat "$RUNTIME_CONFIG" dump is gone; the script now prints only a redacted header count.
  • MEDIUM ($ escaping gap): confirmed fixed. value="${value//\$/\$\$}" was added, correctly neutralizing the Collector's ${env:...} expansion syntax.

Still open in substance (threads already marked resolved on GitHub, so not re-flagged inline below, but noting for visibility):

  • MEDIUM premature-decision (otelcol-contrib header-reading behavior asserted as fact): the code comment is unchanged and no end-to-end verification was added to the PR description; the thread's resolution rests on a reply rather than a code/doc change.
  • MEDIUM (non-ASCII line-break escaping): only ASCII \n/\r are stripped; Unicode line-separator characters are still unhandled.

New findings surfaced during this recheck are below.

Comment thread hack/upload-traces.sh Outdated
Comment thread hack/upload-traces.sh Outdated
Comment thread hack/upload-traces.sh Outdated

@waynesun09 waynesun09 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recheck round 2

Re-reviewed at head 6848df27 (previous recheck was against 2843acf6). Dispatched a 4-agent review squad (independent Claude, Grok, and Codex-backed reviewers) against the current diff; findings below are the deduped, independently cross-verified survivors — each was confirmed by at least 3 of 4 agents and/or reproduced empirically against the real confmap library rather than inferred from reading code alone.

Confirmed resolved since the last recheck:

  • Header secret values no longer persist in cleartext in the generated config file — only header keys and internal REPLAY_HEADER_N variable names ever touch disk now.
  • Duplicate header keys are deduplicated with deterministic last-wins semantics (--header overrides OTEL_EXPORTER_OTLP_HEADERS).
  • The mktemp template no longer has a suffix after XXXXXX, so it randomizes correctly on BSD/macOS mktemp (verified directly — the old template returned a literal, non-randomized path on the first call and hard-failed on every call after; the new one doesn't).
  • The original stdout cleartext leak remains fixed (prints a redacted count only).

Still open / newly found (see inline comments below) — unresolving the premature-decision thread, since the substance persists under the reworded comment:

  • The otelcol-contrib header-reading behavior comment now cites a specific version and method, but is still an unreproducible, unbacked assertion — the PR's own Testing checklist wasn't updated and the tool version pin is still floor-only.
  • The fix that moved header values out of the YAML file text (into per-header env-var indirection) closed the literal-embedding escaping gap but opened a more serious one: the Collector's own config resolver recursively re-expands ${env:...}/${file:...}-shaped text found inside an already-resolved value, so an unsanitized --header/--endpoint value can pivot to exfiltrate an unrelated environment variable or local file. Verified empirically against the real confmap library.
  • A new bash-4-only construct in this commit hard-crashes the entire header feature on stock macOS bash.

Comment thread hack/upload-traces.sh Outdated
Comment thread hack/upload-traces.sh Outdated
Comment thread hack/upload-traces.sh Outdated
Comment thread hack/upload-traces.sh Outdated

@waynesun09 waynesun09 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recheck round 3

Re-reviewed at head afacbac2 (previous recheck was against 6848df27). Dispatched a 4-agent review squad (independent Claude, Grok, and Codex-backed reviewers) against the current diff, then independently verified the security-relevant claims myself against the real go.opentelemetry.io/collector/confmap source and by direct reproduction.

Verified fixed:

  • CRITICAL (confmap recursive re-expansion): genuinely closed. escape_confmap() (line 107) doubles every literal $ to $$ before REPLAY_INCLUDE, REPLAY_ENDPOINT, and every header value are exported. Verified against confmap's actual findURI escape-detection logic (an odd run of $ immediately before ${ is required to trigger expansion; doubling a value's dollar signs always produces an even count, which is left as inert literal text) and against the project's own e2e fixture (confmap/internal/e2e/testdata/expand-escaped-env.yaml), which covers exactly this $${env:...} shape and asserts it resolves to the literal string, never the referenced variable. The >=0.120.0 floor this script requires postdates the $$-escaping fix (landed 0.105.0, stable 0.107.0, unconditional since 0.109.0), so there's no version gap in the supported range.
  • HIGH (declare -A bash 3.2 crash): genuinely closed. No associative arrays remain anywhere in the file; dedup now uses plain indexed arrays with a linear scan, which is bash 3.0+-safe. Confirmed no other bash-4-only syntax was introduced, and confirmed the script runs end-to-end under real stock bash 3.2.57.

Not re-flagged (raised twice, maintainer has explicitly declined both — noting for visibility, not re-litigating): the unverified otelcol-contrib behavior comment, and the bearer-token-shaped usage example.

New findings below — surfaced while tracing the secret data flow end-to-end hunting for a possible 4th regression, independently reproduced, not duplicates of anything raised in prior rounds.

Comment thread hack/upload-traces.sh Outdated
Comment thread hack/upload-traces.sh Outdated
Comment thread hack/upload-traces.sh Outdated
Replace dynamic --header flag parsing with a static config template
that users edit directly. otelcol-contrib reads headers from its YAML
config, not from env vars, so letting users write them in the config
file avoids all the escaping, merging, and injection surface area.

If yq is installed, the script prompts for confirmation when no headers
are configured. Endpoint and include pattern are still passed via env
vars with confmap escaping.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Hector Martinez <hemartin@redhat.com>
@rh-hemartin
rh-hemartin force-pushed the agent/5436-fix-header-passthrough branch from afacbac to b6d72c2 Compare July 24, 2026 07:07
@rh-hemartin

Copy link
Copy Markdown
Member

@waynesun09 I went for a file approach, as the environment parsing was creating so many problems.

@rh-hemartin rh-hemartin changed the title fix(#5436): inject --header values into collector YAML config fix(#5436): move header config to user-edited YAML file Jul 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug ready-for-review Agent PR ready for human review requires-manual-review Review requires human judgment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: upload-traces.sh --header flags silently ignored by otelcol-contrib

2 participants